Systems and Methods for Performing Occlusion Detection

ABSTRACT

The present invention provides a mobile robot configured to navigate an operating environment, that includes a machine vision system comprising a camera that captures images of the operating environment using a machine vision system; detects the presence of an occlusion obstructing a portion of the field of view of a camera based on the captured images, and generate a notification when an occlusion obstructing the portion of the field of view of the camera is detected, and maintain occlusion detection data describing occluded and unobstructed portions of images being used by the SLAM application.

RELATED APPLICATIONS

The present application is a continuation application of and claimspriority from U.S. patent application Ser. No. 14/856,497, filed on Sep.16, 2015, which claims the benefit of and priority from U.S. ProvisionalPatent Application No. 62/085,031, filed on Nov. 26, 2014, entitled“Systems and Methods for Performing Occlusion Detection” and from U.S.Provisional Patent Application No. 62/116,307 filed on Feb. 13, 2015,entitled “Systems and Methods for Performing Simultaneous Localizationand Mapping using Machine Vision Systems”, the disclosures of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

Systems and methods for performing occlusion detection are describedherein

BACKGROUND

Many robots are electro-mechanical machines, which are controlled by acomputer. Mobile robots have the capability to move around in theirenvironment and are not fixed to one physical location. An example of amobile robot that is in common use today is an automated guided vehicleor automatic guided vehicle (AGV). An AGV is typically considered to bea mobile robot that follows markers or wires in the floor, or uses avision system or lasers for navigation. Mobile robots can be found inindustry, military and security environments. They also appear asconsumer products, for entertainment or to perform specific tasks suchas vacuum cleaning and home assistance.

In order to achieve full autonomy, a mobile robot typically needs topossess the ability to explore its environment withoutuser-intervention, build a reliable map of the environment, and localizeitself within the map. Significant research has been conducted in thearea of Simultaneous Localization and Mapping (SLAM) to address thisproblem in mobile robotics. The development of better navigationalgorithms and more accurate sensors have enabled significant progresstowards building better robots.

SUMMARY OF THE INVENTION

The present invention provides a mobile robot configured to navigate anoperating environment, that includes: a body containing: a driveconfigured to translate the robot in a direction of motion; a machinevision system comprising a camera that captures images of the operatingenvironment of the mobile robot; a processor; memory containing a asimultaneous localization and mapping (SLAM) application and abehavioral control application: wherein the behavioral controlapplication directs the processor to: capture images using the machinevision system; detect the presence of an occlusion obstructing a portionof the field of view of the camera based on the captured images; andgenerate a notification when an occlusion obstructing the portion of thefield of view of the camera is detected; and wherein the behavioralcontrol application further directs the processor to maintain occlusiondetection data describing occluded and unobstructed portions of imagesbeing used by the SLAM application.

In some embodiments, the occlusion detection data describes portions ofimages that correspond to portions of the field of view of the camera.

In several embodiments, the occlusion detection data includes ahistogram that identifies different portions of the camera field of viewand provides a corresponding frequency with which each portion of thefield of view is used by the SLAM application to generate or identifyfeatures.

In certain embodiments, detecting the presence of an occlusion furtherincludes identifying a portion of the camera field of view in thecaptured images that is not utilized by the SLAM application for athreshold number of images.

In a number of embodiments, detecting the threshold number of images isbetween at least 1-10 images per 300 mm of robot travel for a mobilerobot traveling at a rate of speed between 220 mm per second and 450 mmper second.

In some embodiments, the behavioral control application further directsa processor to identify a certain percentage of the camera field of viewis occluded prior to generating the notification.

In some embodiments, the occluded percentage of the camera field of viewis between 30 and 75 percent.

In some embodiments, the notification is one of a displayed message,text message, voicemail, or electronic mail message.

In several embodiments, the notification is one of an alert, sound, orindicator light provided by the mobile robot.

In some embodiments, the behavioral control application further directsa processor to send the notification of the occlusion to a server.

In certain embodiments, the occlusion is detected within a first portionof a set of captured images.

In several embodiments, the occlusion is a semi-transparent occlusionthat blurs the first portion of the set of captured images thatcorresponds to the obstructed portion of the field of view.

In a number of embodiments, the occlusion is an opaque occlusion thatcompletely occludes the first portion of the set of captured images thatcorresponds the obstructed portion of the field of view.

In some embodiments, the camera is positioned within a recessedstructure under the top surface of the mobile robot and is tilted sothat the camera optical axis is aligned at an acute angle of 30-40degrees above a horizontal plane in line with the top surface and isaimed in a forward drive direction of the robot body, and wherein thefield of view of the camera spans a frustum of 45-65 degrees in thevertical direction.

In some embodiments, the camera is protected by a lens cover aligned atan acute angle with respect to the optical axis of the camera.

In some embodiments, the lens cover is set back relative to an openingof the recessed structure and an acute angle with respect to the opticalaxis of the camera that is closer to perpendicular than an angle formedbetween a plane defined by the top surface and the optical axis of thecamera.

In some embodiments, the acute angle is between 15 and 70 degrees.

In some embodiments, the angle formed between a plane defined by theopening in the recessed structure and the optical axis of the cameraranges between 10 and 60 degrees.

Some embodiments of the invention provide a mobile robot configured tonavigate an operating environment, that includes: a body containing: adrive mechanism configured to translate the robot in a direction ofmotion; an odometry sensor system; a machine vision system comprising acamera that captures images of the operating environment of the mobilerobot; and at least one processor; memory containing a behavioralcontrol application and a visual simultaneous localization and mapping(VSLAM) application, a pre-loaded map of the operating environment, andan estimated robot pose; wherein the behavioral control applicationdirects a processor to: actuate the drive and capture: odometry datausing the odometry sensor system; and images using the machine visionsystem; determine updated robot poses and update the map of theoperating environment by providing at least the captured images and theodometry data as inputs to the VSLAM application; update occlusiondetection data describing portions of images being used by the VSLAMapplication to determine updated robot poses and update the map of theoperating environment; detect the presence of an occlusion obstructingthe camera based on the occlusion detection data; and provide anotification when the occlusion detection data indicates the presence ofthe occlusion obstructing the camera, wherein the VSLAM applicationdirects a processor to: estimate the location of the mobile robot withinthe map of the environment based upon images captured by the machinevision sensor system and odometry data captured by the odometry sensorsystem, and update the map based upon the images captured by the machinevision sensor system.

In some embodiments, the occlusion detection data describes portions ofimages that correspond to portions of a field of view of the camera.

In several embodiments, the occlusion detection data includes ahistogram that identifies different portions of the camera field of viewand provides a corresponding frequency with which each portion of thefield of view is used by the VSLAM application to generate or identifyfeatures.

In a number of embodiments, detecting the presence of an occlusionobstructing the camera based on the occlusion detection data furtherincludes identifying a portion of the camera field of view in thecaptured images that is not utilized by the VSLAM application for athreshold number of images.

In certain embodiments, the threshold number of images is between atleast 1-10 images per 300 mm of robot travel for a mobile robottraveling at a rate of speed between 220 mm per second and 450 mm persecond.

In several embodiments, detecting the presence of an occlusionobstructing the camera based on the occlusion detection data furtherincludes identifying a portion of the camera field of view capturingimage data with pixel characteristics having a threshold level ofsimilarity.

In certain embodiments, the behavioral control application furtherdirects a processor to identify a certain percentage of the camera fieldof view is occluded prior to providing the notification.

In a number of embodiments, the notification is one of a displayedmessage, text message, voicemail, or electronic mail message.

In several embodiments, the notification is one of an alert, sound, orindicator light provided by the mobile robot.

In certain embodiments, the VSLAM application further directs aprocessor to: identify features within different portions of a capturedimage; and update the occlusion detection data describing portions ofimages being used by the VSLAM application based on the portions of thecaptured image that contain features.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a front perspective view of a mobile robot.

FIGS. 1B illustrates a bottom view of a mobile robot.

FIG. 1C illustrates a block diagram of a controller circuit of a robot.

FIG. 2A illustrates a top perspective view of a mobile robot thatincludes a tilted camera within a recessed structure.

FIG. 2B illustrates a cross-sectional view of a tilted camera of amobile robot.

FIGS. 3A-3C illustrate a lens holder that prevents lens deformationsthat may occur in the camera lens due to transfer of mechanical stressesfrom a mobile robot to the lens.

FIG. 4 illustrates a mobile robot configured with forward and backwardfacing tilted cameras contained within separate recesses within the topof the body of the mobile robot and protected using lens covers.

FIG. 5 illustrates a cross-sectional view of the front and rear facingtilted cameras of a mobile robot configured in the manner illustrated inFIG. 4.

FIG. 6 illustrates a mobile robot controller circuit that can be usedfor VSLAM using an enhanced navigation system.

FIG. 7 conceptually illustrates a mobile robot behavioral controlapplication configured to enable navigation within an environment basedupon (but not limited to) a VSLAM process.

FIG. 8 is a flow chart illustrating a process that can optionally beused by a mobile robot to navigate an environment.

FIG. 9 is a flow chart illustrating a process that can optionally beutilized by the mobile robot to identify a new landmark for use innavigation.

FIG. 10 is a flow chart illustrating a process that can optionally beutilized by a mobile robot to determine relative pose using previouslycreated landmarks.

FIGS. 11A-11C illustrate examples of a field of view captured by amobile robot that has a camera configured such that the optical axis isaligned with the forward direction of motion of the mobile robot.

FIGS. 12A-12C illustrate an example of views of a scene captured by acamera tilted so that its optical axis forms an acute angle above thedirection of motion of a mobile robot.

FIG. 13A illustrates disparity that can be utilized to determinedistance to a feature.

FIG. 13B illustrates an example of upward movement of features todetermine disparity.

FIG. 14A illustrates an image generated by simulating a viewpoint of aforward facing camera of a simulated mobile robot in a virtual indoorenvironment

FIG. 14B illustrates an image generated by simulating a viewpoint of acamera having the same field of view as the camera used in thesimulation used to generate the image shown in FIG. 14A with theexception that the camera is tilted so that the optical axis of thecamera is at a 30 degree angle above the direction of forward motion ofthe simulated mobile robot

FIG. 15 is a chart showing a comparison of the precisions with which amobile robot is able to determine its relative pose in the floor planein each of the simulations described above.

FIG. 16A illustrates a sampling grid of a camera of a mobile robotconfigured using a wide angle lens.

FIG. 16B illustrates sampling a scene using a tilted camera having anarrow field of view.

FIGS. 17A illustrates examples of specific types of occlusions and theresulting images captured by the occluded camera.

FIG. 17B illustrates an example of an image captured by a camera inwhich the semi-transparent occlusion results in a blurred portion of theimage.

FIG. 18A illustrates an example of an opaque occlusion.

FIG. 18B illustrates an example of an image captured by a camera inwhich the opaque occlusion results in complete occlusion of a portion ofa scene.

FIG. 19 is a flow chart illustrating an occlusion detection process thatcan optionally be performed by a mobile robot.

FIG. 20 illustrates a communication diagram illustrating communicationbetween a mobile robot, an external server and a user device.

FIG. 21 illustrates a system for notifying user devices of an occlusion.

FIG. 22 illustrates a robot having a top surface that is not more than 4inches from the floor surface, a camera mounted under the top surface ofthe mobile robot, having a field of view spanning a frustum having afield of view angle δ of approximately 50-60 degrees in the verticaldirection and an optical axis 155 at an acute angle φ of approximately30 degrees above horizontal.

DETAILED DESCRIPTION

Turning now to the drawings, particularly the FIGS. 1A-1C, 4 and 5,systems and methods for performing Visual Simultaneous Localization andMapping (VSLAM) using mobile robot 10 machine vision systems includingone or more tilted cameras 125 are illustrated. In implementations, theone or more tilted cameras 125 are recessed within a central portion ofthe body 100 of the mobile robot 10 at a fixed angle relative to the topsurface 108 of the robot and each of the one or more tilted cameras 125are protected by a lens cover 135 aligned at an acute angle with respectto the optical axis 155 of the camera. Where a camera 125 is recessed ina portion of the body 100 of the mobile robot 10, the lens cover 135 mayalso be set back relative to the opening of the recess 130. Inimplementations, the navigation system 120 is part of a mobile robot 10that is a house cleaning robot no taller than 4 inches from a floorsurface to the top surface 108 of the robot. In this way, the housecleaning robot 10 can navigate into tight spaces (e.g., beneath chairsand beneath the face frame of kitchen cabinetry). In severalimplementations, the mobile robot 10 is 13.9 inches in diameter andweighs 8.4 lbs and moves at a speed of 220 mm per second to 450 mm persecond. In a still further optional aspect of the invention, the one ormore tilted cameras 125 are tilted so that the house cleaning robot 10can capture images of reliably static, feature rich objects, such aspicture frames hung on the walls of a home and other features having nodisplacement. As is discussed further below, the mobile robot 10 can usefeatures of reliably static objects located at a particular range ofheights above the floor to build a map of an environment and navigateusing vision based sensors and vision based simultaneous localizationand mapping, or VSLAM.

The combination of SLAM with visual sensors is often referred to asVSLAM 740. VSLAM 740 processes are typically vision and odometry-based,and enable reliable navigation in feature rich environments. Such visualtechniques can be used by a vehicle, such as a mobile robot 10, toautonomously navigate an environment using a self-generated map that iscontinuously updated. A variety of machine vision systems have beenproposed for use in VSLAM 740 including machine vision systems thatinclude one or more cameras 125.

Systems and methods in accordance with embodiments of the inventionperform VSLAM 740 using a camera 125 mounted under the top surface 108of a mobile robot 10 and having an optical axis 155 aligned at an acuteangle above the top surface 108 of a mobile robot 10. Many VSLAM 740processes analyze disparity of features captured in a series of imagesin order to estimate distances to features and/or triangulate position.In general, the amount of disparity observed for a set of featuresbetween a series of images that have been captured from differentvantage points determines the precision with which these features aremapped within the environment. The greater the observed disparity, themore accurate the distance measurement.

When a mobile robot 10 employs a forward looking camera 125 having anoptical axis 155 of the lens 140 aligned parallel with the direction offorward motion, there is generally only a minimal amount of disparitythat can be observed in a set of features positioned directly in frontof the mobile robot 10 as the robot moves toward the features. As isdiscussed further below, features visible in the center of the field ofview of a forward looking camera are likely to increase in scale as amobile robot 10 moves toward the feature, with little discernabledisparity between successive images of the feature. Accordingly, 3Dstructure of features within the center of the field of view of aforward looking camera may be difficult to ascertain from a series ofimages captured as the mobile robot 10 moves forward toward thefeatures. The problem is particularly acute for small mobile robots,such as house cleaning robots, where the robot form factor dictatescamera placement close to the ground (e.g., 4 inches or less above theground). Location precision of forward looking cameras can be improvedby increasing the field of view of the forward looking camera. However,increasing field of view reduces the angular resolution of the imagedata captured for a given image sensor resolution. Furthermore, thefield of view is increased so that disparity can be observed withrespect to the peripheral portions of the field of view, or the off-axisfield of view, of the camera where the distortion of wide angle lensesis typically greatest and the angular resolution of the images capturedby the camera is lowest.

Tilting a camera at an angle above the forward direction of motion canincrease the disparity observed across the field of view of the camerawhen the camera moves toward an object. In particular, tilting thecamera at an angle above the forward direction of motion increases thedisparity observed within the center of the field of view, the portionof the camera field of view with the highest angular resolution. Afeature observable within the center of the field of view of a cameratilted so that its optical axis 155 forms an acute angle relative to thehorizontal axis aligned with the forward direction of motion will moveupward in a series of images captured while moving toward the feature.Accordingly, objects such as (but not limited to) picture frames andtelevisions that are frequently hung on the walls of residences andhaving readily discernable features at any scale and in various lightingconditions, provide excellent navigation landmarks for mobile robotsthat operate in residential environments, such as house cleaning robots.

Tilting the camera 125 upward can also enable the mobile robot 10 tomore precisely determine the 3D structure of the underside of objectshung on walls. Furthermore, tilting the camera 125 allows the mobilerobot 10 to focus on an area within a typical indoor environment inwhich features are unchanging, such as those features imaged around doorframes, picture frames and other static furniture and objects, allowingthe mobile robot 100 to identify reliable landmarks repeatedly, therebyaccurately localizing and mapping within an environment. Additionally,in implementations, the camera 125 on the mobile robot 10 is a 320×240QVGA, 0.0768 mP camera (or 640×480 VGS, 0.3MP camera) transferringimages at a rate of less than 30 milliseconds with an image processingrate of 3 frames per second. In implementations, the camera 125 has noIR filter for better detection of features in low illuminationenvironments. In implementations, the mobile robot 10 will create newlandmarks if the number of detectable features falls below a thresholdfor a minimum number of features for detecting a landmark. Inembodiments, the threshold number of landmarks is a cluster ofidentifiable features detected at a rate 1-10 landmarks per foot oftravel and preferably 3 landmarks per foot of robot travel at a rate ofapproximately 1 ft per second or approximately 306 mm per second. Inimplementations, if the environment is too dark and the illumination isinsufficient for feature detection, the mobile robot 10 will depend onanother sensor, such as an optical dead reckoning sensor aimed at thefloor (e.g. an LED or laser illuminated mouse sensor) to localize.

Disparity measurements made with respect to features visible within thecenter of the field of view of the camera 125 can be made more preciselyby sacrificing field of view for increased angular resolution. Inseveral embodiments, the angular resolution achieved using a specificsensor resolution is increased relative to a forward looking cameraemploying a wide angle lens by utilizing a tilted camera 125 thatemploys a view lens 140 having a horizontal field of view of, forexample, 78 degrees. A typical distinction between narrow field of viewlenses 140 and wide angle lenses is that perspective projection isgenerally a good approximation to the true imaging characteristics ofnarrow field of view lenses, whereas wide angle lenses introducedistortion. Accordingly, the angular resolution and modulation transferfunction (MTF) of a lens tends to be more even over the field of view ofa narrow field of view lens when compared to the variation in angularresolution and MTF experienced across the field of view of a wide anglelens. The distortion introduced by a wide angle lens can be correctedusing computationally expensive distortion correction operations anddisparity can be determined following rectification of the imagescaptured by the camera. When a narrow field of view camera 125 isutilized, unrectified images can be utilized in subsequent imageprocessing processes. Accordingly, use of narrow fields of view lensesin combination with appropriate VSLAM processes can provide advantagesrelative to more traditional VSLAM processes using wide field of viewlenses in that they avoid performing the additional computationallyexpensive step of rectification and can locate features and/or landmarks(i.e. groups of features) with higher precision. An example of animplementation using a narrow field of view lens to capture anenvironment is described in detail below with reference to FIG. 16A.

Many VSLAM processes rely upon having a wide baseline between observedfeatures to perform localization. As is discussed further below, analternative approach involves identifying related features havingdistinctive 3D structure as landmarks. Relative pose of a mobile robot10 can be readily determined based upon the 3D structure of the featuresforming a landmark and the appearance of the features in an imagecaptured by the mobile robot 10. Therefore, the precision of a relativepose estimate is largely dependent upon the angular precision with whichthe distances between features observed within an image captured by themobile robot's 10 camera 125 can be measured. Accordingly, configuringthe mobile robot 10 with a tilted camera 125 having a narrow field ofview lens can achieve improved performance over the equivalent systemequipped with a forward facing camera or a camera with a wider field ofview and the same resolution. Implementations using a tilted camera 125are described in detail below with reference to FIGS. 1A and 2A-B.

In many embodiments, the field of view of a mobile robot's 10 camera 125and the specific angle formed between the camera's optical axis 155 andthe forward direction of motion is determined based upon therequirements of specific applications including (but not limited to) theframe rate of the camera 125, the speed of the mobile robot 10, and theprocessing capacity of the processor(s) utilized to perform imageprocessing within the mobile robot 10. As can readily be appreciated,the greater the speed of the mobile robot 10 the greater the observeddisparity between captured frames. Disparity is observed by overlappingthe FOV of frames, and the mobile robot 10 moves at a speed of between220 mm per second and 450 mm per second and preferably travels at aspeed of 306 mm per second. Increasing the tilt of the camera 125 canfurther increase the disparity. The ability to observe the disparitydepends upon the rate at which the mobile robot 10 can capture andprocess frames of image data in real time. A lower frame rate can becompensated for by a larger field of view. As noted above, however,increasing field of view can come at the computational cost ofrectifying the captured images. Therefore, in many optionalconfigurations of the mobile robot 10, camera 125 tilt and field of vieware selected to meet the requirements of a specific mobile robot and itsoperating environment.

The effective field of view of the navigation system 120 can beincreased without decreasing angular resolution by adding additionalcameras 125, 410 positioned at different locations around the mobilerobot 10. In one optional aspect of the invention, each camera 125, 410is housed within a separate recess 130 in the body 100 of the mobilerobot 10 with a separate protective lens cover 135. Certain embodimentsmay include both a front facing tilted camera 125 and one or more rearfacing tilted cameras 410. Implementations using multiple cameras aredescribed in detail with reference to FIGS. 4 and 5 below. The use ofseveral cameras 125, 410 may allow the mobile robot 10 to observe morefeatures and/or landmarks for use in VSLAM while maintaining an angularprecision with which these features are detected. Readily identifyinglandmarks in one or more directions around the robot 10 assists withquickly re-localizing to resume a cleaning mission and continue to builda map of the environment following a kidnapping event in which the robot10 is moved or lost. Furthermore, a rear facing tilted camera 410 mayallow the VLSAM process to use the same landmarks irrespective of thedirection of motion. This can be useful in a mobile robot 10 thatnavigates an environment in a “corn-row” pattern. In this situation,every time the mobile robot 10 turns around, it can use features and/orlandmarks that were visible to the forward facing camera 125 to navigateon the return path using image data captured by the rear facing camera410. Furthermore, the rear facing tilted camera 410 may detect thelandmarks with a higher angular precision than if the mobile robot 10were to use, for example, a 360 degree omnidirectional camera given theincreased spatial resolution of the narrower field of view lens(assuming comparable sensor resolution).

During operation, the one or more cameras 125 mounted on the mobilerobot 10 may become obstructed for any of a variety of reasons. Forexample, dust and debris may accumulate on the camera lens 140, or alens cover 135, over time and with use of the mobile robot 10 and thusocclude portions of the images being captured. The mobile robot 10 isable to detect when some type of obstruction is occluding one or moreportions of a camera lens 140. When an occlusion is detected, the mobilerobot 10 may provide a notification that notifies a user to, forexample, clean a lens cover 135 protecting the camera lens 140 in orderto remove the occlusion. In order to detect the presence of an occlusionobstructing a portion of the field of view of a camera 125, someembodiments analyze the particular portions of the images providinguseful information for a VSLAM process, and based on this analysis, areable to determine that certain other portions of the field of view maybe occluded. In particular, some embodiments may maintain a histogram ofthe different portions of the field of view and a frequency with whicheach portion is capturing image data that is being used to generate newlandmarks and/or recognize existing landmarks during the navigation of amobile robot 10 through an environment using VSLAM. Regions that areused with low frequency can be flagged as occlusions and notificationsgenerated accordingly.

Although much of the discussion that follows describes cameraconfigurations used in combination with specific VSLAM processes, thetechniques disclosed herein can be utilized by the mobile robot 10configured using any of a variety of different mapping and navigationmechanisms. Accordingly, various optional configurations of the mobilerobot 10 incorporating one or more tilted cameras 125 for use innavigating an environment are discussed further below.

Mobile Robots with Enhanced Vision Sensor Configurations

The mobile robot 10 incorporates a navigation system 120 including acamera 125 that can capture image data used by a VSLAM processes in thenavigation of the mobile robot 10 and the mapping of the environmentsurrounding the mobile robot 10. The tilted camera 125 used in thenavigation system 120 of the mobile robot 10 is illustrated in FIGS.1A-2B. In particular, FIG. 1 illustrates front perspective view of themobile robot 10 and FIG. 2 illustrates the recess 130 disposedsubstantially in the middle of the body 100 of the mobile robot 10containing the tilted camera 125. As can readily be appreciated, thecamera 125 is protected by a lens cover 135 that is configured so thatone edge of the lens cover 135 is closer to the opening of the recess130 than a second edge of the lens cover 135. In this way, the lenscover 135 forms an acute angle α with the optical axis 155 of the camera125 that is closer to perpendicular than the angle φ formed between aplane defined by the opening in the recessed structure 130 within thebody 100 of the mobile robot 10 and the optical axis 155 of the camera125. In some embodiments, the acute angle α may range between 15 and 70degrees and the acute angle φ may range between 10 and 60 degrees.Angling the lens cover 135 in this way is an optional aspect of theinvention and the lens cover 135 can be implemented in the plane of theopening in the recess 130 of the mobile robot body 100 containing thetilted camera 125 or recessed, but parallel to the plane of the openingin the recess 130 of the mobile robot body 100. The configuration of thetilted camera 125 within the recess 130 in the body 100 of the mobilerobot 10 is discussed further below.

As shown in FIGS. 1B showing the bottom of the robot 10 and FIG. 1Cdepicting a block diagram of a controller circuit 605 of the robot 10and systems of the robot 10 operable with the controller circuit 605,the mobile robot 10 includes a body 100 supported by a drive 111 locatedbeneath the body 100 including left and right drive wheel modules 111 a,111 b, that can maneuver the robot 10 across a floor surface. Inimplementations, the drive is that of the robot described in US PatentApplication Publication No. 2012/0317744, herein incorporated byreference in its entirety. In many implementations, the mobile robot 10is no taller than 4 inches from a floor surface in order to allow themobile robot 10 to navigate through tight spaces within a typical indoorenvironment.

The mobile robot 10 can be configured to actuate its drive 111 based ona drive command. In some embodiments, the drive command may have x, y,and θ components and the command may be issued by a controller circuit605. The mobile robot body 100 may have a forward portion 105corresponding to the front half of the shaped body, and a rearwardportion 110 corresponding to the back half of the shaped body. The driveincludes right and left driven wheel modules 111 a, 111 b that mayprovide odometry to the controller circuit 605. The wheel modules 111 a,111 b are substantially disposed along a transverse axis X defined bythe body 100 and include respective drive motors 112 a, 112 b drivingrespective wheels 113 a, 113 b. The drive motors 112 a, 112 b mayreleasably connect to the body 100 (e.g., via fasteners or tool-lessconnections) with the drive motors 112 a, 112 b optionally positionedsubstantially over the respective wheels 113 a, 113 b. The wheel modules111 a, 111 b can be releasably attached to the chassis and forced intoengagement with the cleaning surface by springs. The mobile robot 10 mayinclude a caster wheel 116 disposed to support a portion of the mobilerobot body 100, here, a forward portion of a round body 100. In otherimplementations having a cantilevered cleaning head, such as a squarefront or tombstone shaped robot body 100, the caster wheel is disposedin a rearward portion of the robot body 100. The mobile robot body 100supports a power source (e.g., a battery 117) for powering anyelectrical components of the mobile robot 10.

Referring again to FIGS. 1A and 1B, the mobile robot 10 can move acrossa cleaning surface through various combinations of movements relative tothree mutually perpendicular axes defined by the body 100: a transverseaxis X, a fore-aft axis Y, and a central vertical axis Z. A forwarddirection of motion along the fore-aft axis Y is designated F (sometimesreferred to hereinafter as “forward”), and an aft drive direction alongthe fore-aft axis Y is designated A (sometimes referred to hereinafteras “rearward”). The transverse axis X extends between a right side R anda left side L of the robot substantially along an axis defined by centerpoints of the wheel modules.

In many embodiments, a forward portion 105 of the body 100 carries abumper 115, which can be utilized to detect (e.g., via one or moresensors of the bumper sensor system 550) events including (but notlimited to) obstacles in a drive path of the mobile robot 10. Dependingupon the behavioral programming of the mobile robot 10, it may respondto events (e.g., obstacles, cliffs, walls) detected by the bumper 115,cliff sensors 119 a-119 f, and one or more proximity sensors 120 a-120 nby controlling the wheel modules 111 a, 111 b to maneuver the robot 10in response to the event (e.g., back away from a detected obstacle).

As illustrated, a user interface 126 is disposed on a top portion of thebody 100 and can be used to receive one or more user commands and/ordisplay a status of the mobile robot 10. The user interface 126 is incommunication with the controller circuit 605 carried by the robot 10such that one or more commands received by the user interface 126 caninitiate execution of a cleaning routine by the robot 10.

The mobile robot 10 may also include a camera 125 navigation system 120embedded within the body 100 of the robot 10 beneath the top cover 108.The navigation system 120 may include one or more cameras 125 (e.g.,standard cameras, volumetric point cloud imaging cameras,three-dimensional (3D) imaging cameras, cameras with depth map sensors,visible light cameras and/or infrared cameras) that capture images ofthe surrounding environment. In one optional configuration, the camera125 captures images of the environment that are positioned at an acuteangle relative to the axis of motion (e.g., F or A) of the mobile robot10. For example, as illustrated in FIG. 22, in implementations of arobot 10 having a top surface 108 that is not more than 4 inches fromthe floor surface, a camera 125 mounted under the top surface 108 of therobot 10, having a field of view spanning a frustum of approximately50-60 degrees in the vertical direction and an optical axis 155 angledat approximately 30 degrees above horizontal will detect features in theenvironment at a height of generally 3-14 feet. For example, a mobilerobot 10 of these dimensions with these camera settings will see objectsat a height of approximately 6 inches to 4.5 feet at a distance of 3feet, at a height of approximately 9 inches to 7.5 feet at a distance of5 feet and at a height of approximately 1.2 feet to 14 feet at adistance of 10 feet. By focusing the undistorted central portion of thecamera 125 field of view on an area in which features are unchanging,such as those features imaged around door frames, picture frames andother static furniture and objects, the robot 10 identifies reliablelandmarks repeatedly, thereby accurately localizing and mapping withinan environment.

In these embodiments, the lens 140 (FIGS. 2A and 2B) of the camera 125is angled in an upward direction such that it primarily captures imageshaving their reliable, undistorted portions focused on feature rich,unchanging areas of the walls, wall-ceiling interfaces, and a portion ofthe ceiling surrounding the mobile robot 10 in a typical indoorenvironment. As noted above, many environments including (but notlimited) residential housing environment include varied and staticobjects hung and clustered on walls that provide features and/orlandmarks that are useful for performing navigation. Objects typicallyavailable in the region of approximately 2-15 feet (e.g., 3-10 feet, 3-9feet, 3-8 feet), such as door and window frames, pictures and largefurniture, have little displacement therefore provide reliably imagedfeature geometry that results in the creation of more accurately andreliably identified landmarks. By aggregating reliable features in thisregion in a narrow field of view, determination of landmarks improvesand localization determination therefore improves. By aggregating variedand distinguished features imaged from items having no displacement, themobile robot 10 builds a reliable map of landmarks.

The camera 125 may optionally be tilted so that the lower periphery ofthe field of view of the camera 125 is unoccluded by the body 100 of themobile robot 10. Alternatively, in implementations, the body 100 of themobile robot 10 partially occludes a lower portion of the field of viewof the tilted camera 125 and the controller circuit 605 discards thisportion of the field of view when imaging features. As noted above,tilting the camera 125 can increase the amount of disparity observedacross the field of view of the camera 125 as the mobile robot 10 movesthrough the environment. In an implementation, the mobile robot 10employs a titled camera 125 with optics having a sufficiently narrowfield of view so that perspective projection can be assumed to be a goodapproximation to the true imaging characteristics of the narrow field ofview lens. Subsequent image processing can be performed withoutrectifying the images and the camera 125 can observe features with ahigher angular resolution than a wider angle lens that would alsointroduce distortion.

The images captured by the camera 125 may be used by VSLAM processes inorder to make intelligent decisions about actions to take to maneuverthe mobile robot 10 about an operating environment. While the camera 125of the navigation system 120 is illustrated in FIGS. 1A-1C and 2A-2B asbeing contained within a centrally located recess 130 under the topsurface 108 of the mobile robot 10, cameras 125 forming part of themobile robot's 10 navigation system 120 can additionally oralternatively be arranged at any of one or more positions andorientations on the mobile robot 10, including on or within the frontbumper and along the sides of the mobile robot.

In addition to the camera 125 of the navigation system 120, the mobilerobot 10 may include different types of sensor systems 500 in order toachieve reliable and robust autonomous movement. The additional sensorsystems 500 may be used in conjunction with one another to create aperception of the mobile robot's 10 environment sufficient to allow therobot to make intelligent decisions about actions to take in thatenvironment. The various sensor systems may include one or more types ofsensors supported by the robot body 100 including, but not limited to,obstacle detection obstacle avoidance (ODOA) sensors, communicationsensors, navigation sensors, range finding sensors, proximity sensors,contact sensors (e.g. bumper sensors), sonar, radar, LIDAR (LightDetection And Ranging, which can entail optical remote sensing thatmeasures properties of scattered light to find range and/or otherinformation of a distant target), and/or LADAR (Laser Detection andRanging). In some implementations, the sensor system includes rangingsonar sensors, proximity cliff detectors 119 a-119 f, proximity sensors120 a-120 n (e.g., “n” being an unlimited number in an array ofproximity sensors looking out the sidewalls of the robot 10) contactsensors in the bumper sensor system 550, a laser scanner, and/or animaging sonar.

There are several challenges involved in placing sensors on a roboticsplatform. First, the sensors are typically placed such that they havemaximum coverage of areas of interest around the mobile robot 10.Second, the sensors are typically placed in such a way that the robotitself causes an absolute minimum of occlusion to the sensors; inessence, the sensors should not be placed such that they are blinded bythe robot itself. Third, the placement and mounting of the sensorsshould not be intrusive to the rest of the industrial design of theplatform. In terms of aesthetics, it can be assumed that a robot withsensors mounted inconspicuously is more attractive than otherwise. Interms of utility, sensors should be mounted in a manner so as not tointerfere with normal robot operation (e.g., snagging on obstacles).

Additional options that can be employed in the implementation of thenavigation system 120 of the mobile robot 10 are discussed furtherbelow.

Machine Vision System

In order to navigate through an environment, the mobile robot 10 may useinformation gathered from various different types of sensors in order toascertain the characteristics of its surrounding environment. As notedabove, the mobile robot 10 uses a navigation system 120 that includesone or more cameras 125 that capture images of the surroundingenvironment. The images may be provided to a VSLAM process for use inthe localization and mapping of the mobile robot 10 within theenvironment. FIG. 2A illustrates a top perspective view and FIG. 2Billustrates a cross-sectional view of the tilted camera 125 of themobile robot 10 illustrated in FIG. 1A. In particular, FIG. 2A (andcorresponding FIG. 2B) illustrates a tilted camera 125 housed within arecessed structure 130 in the body 100 of the mobile robot 10 andcovered by a lens cover 135. As depicted in FIGS. 2B and 22, the camera125 includes a camera lens 140 with an optical axis 155 that is at anacute angle φ relative to horizontal axis defining the direction ofmotion of the mobile robot 10. As such, the lens 140 is primarily aimedin a direction that will capture the walls, the wall-ceilingintersections and, to a lesser extent the, ceilings within a typicalindoor environment. For example, as illustrated in FIG. 22, inimplementations of a robot 10 having a top surface 108 that is not morethan 4 inches from the floor surface 2205, a camera 125 mounted underthe top surface 108 of the mobile robot 10, having a field of viewspanning a frustum having a field of view angle δ of approximately 50-60degrees in the vertical direction and an optical axis 155 at an acuteangle φ of approximately 30 degrees above horizontal will detectfeatures 2215 in the environment at a height of generally 3-14 feet. Forexample, a mobile robot 10 of these dimensions with these camerasettings will see objects at a height of approximately 6 inches to 4.5feet at a distance Dw of 3 feet, at a height of approximately 9 inchesto 7.5 feet at a distance Dw of 5 feet and at a height of approximately1.2 feet to 14 feet at a distance Dw of 10 feet. By focusing theundistorted field of view of a tilted camera 125 on a feature rich areain which features are unchanging, such as those features imaged at doorframes, picture frames 2215 and other static furniture and objects withreadily imaged feature geometry, the robot 10 identifies reliablelandmarks repeatedly, thereby accurately and efficiently localizing andmapping within an environment. This is particularly useful forre-localizing after a kidnapping event that interrupts the robot's 10mission. A robot 10 that has been moved from one location to another orthat enters a dark room or under furniture area quickly identifies aunique set of features at these heights and knows its location readilyand with exactitude. A camera aimed at an unchanging ceiling or aceiling with repetitive features, such as a dropped ceiling with tilesor a ceiling with evenly spaced lighting features, will not readily knowits location and will have to seek out unique features on which tolocalize, such as a corner of a room. The mobile robot 10 of the presentinvention therefore is efficient and accurately picks up its cleaningroutine at a useful location without having to move randomly to an areawhere a landmark is recognizable on the map.

In the implementation of FIG. 22, the field of view of the camera 125hits a vertical planar surface, such as a household wall, at an upperangle β of 35 degrees and lower angle Ψ of 95 degrees and the opticalaxis 155 intersects the vertical planar surface at a lower angle τ of 60degrees. Wall features become useful far sooner as they are approachedwith a camera 125 having a smaller field of view. For example, inimplementations, the robot 10 includes such a tilted camera 125 as thatdescribed immediately above aimed at a wall 2210 and having a field ofview of 50 degrees, which equates to 3 times the resolution of adirectly forward looking camera (e.g., at a distance from the wall Dw of8 ft, for directly ahead features about 12 pixels per inch, and for 8foot high features, about 6 pixels per inch). The features with thehighest resolution will be the lowest features on the wall, and so thecamera 125 is aimed so as not to miss those features. The configurationof the camera 125 of FIG. 22 creates more trackable optical flow butmaintains a high pixel count by having at least half of the field ofview above 45 degrees. Features at 45 degrees from the robot 10 that inthe field of view of the camera 125 will instantaneously track up thevery middle of the camera 125 field of view at the same speed as themobile robot 10. In embodiments, the mobile robot 10 moves at a speed ofbetween 220 mm per second and 450 mm per second and preferably moves ata speed of approximately 306 mm per second, or 1 ft for second. As themobile robot 10 advances toward the wall 2210, all objects will grow inthe field of view. Features lower than 45 degrees will track more slowlyand accelerate up and sideways than those at 45 degrees, and featureshigher than 45 degrees will track faster and accelerate up more quicklythan those at 45 degrees. In implementations, the features lower than 45degrees will track slower than approximately 1 ft per second andfeatures higher than 45 degrees will track faster than 1 ft per second.In implementations, a first set of edge pixels track vertically at arate that appears faster than the ego-motion of the mobile robot 10 anda second set of edge pixels tracks vertically at a rate that appearsequal to or slower than the robot ego-motion. In implementations, thecamera 125 is arranged less than 6 inches from the floor, with a 50degree field of view directed to span a 50 degree range limited between−10 degrees and 90 degrees above the horizon (e.g., extending from 30-80degrees), and the camera 125 is directed to be limited to viewingportions of walls forward of the mobile robot 10 at a height of between3 and 8 feet as long as the mobile robot 10 is a distance Dw of between3-10 feet from the wall 2210.

Returning to the implementation of FIGS. 2A and 2B, the lens cover 135is also depressed into the recessed structure 130 and is positionedunder the top surface 108 of the mobile robot 10. In addition, the lenscover 135 is aligned at an acute angle α relative to the optical axis155 of the camera, which is greater than the acute angle φ between theplane forming the opening of the recessed structure 130 and the opticalaxis 155 of the camera 125. In embodiments, the acute angle α rangesbetween 15 and 70 degrees and the acute angle φ ranges between 10 and 60degrees. Angling the lens cover 125 relative to the optical axis 155prevents unwanted imaging problems, such as light reflections and/orrefractions that might block the camera 125 from effective imaging offeatures.

As noted above, the mobile robot 10 can optionally include a narrowfield of view lens 140 that provides images in which perspectiveprojection can be assumed to be a good approximation to the true imagingcharacteristics of narrow field of view lenses. Where a narrow field ofview lens 140 is utilized by the mobile robot 10, the transfer ofmechanical stresses from the mobile robot 10 to the lens 140 can distortthe lens eliminating some of the benefits of utilizing the narrow fieldof view lens 140 by introducing a complex distortion correction processas part of the image processing pipeline. The design of the lens holdercan play an important role in preventing the transfer of mechanicalstresses from the mobile robot 10 to the lens 140 and avoidingdistortion of the lens 140. Implementations of lens holders that canoptionally be utilized in camera(s) 125 of the mobile robot 10 arediscussed further below.

Lens Holders

FIGS. 3A-3C depict an implementation of a lens holder 310 that preventslens deformations that may occur in the camera lens 140 due to transferof mechanical stresses from the mobile robot 10 to the lens 140. FIG. 3Aillustrates a lens holder 310 for holding a camera lens 140. The lensholder 310 is connected to a set of screw bosses 320 supported by a setof spider ribs 330. FIG. 3B illustrates a side view of the lens holder310. In particular, this figure illustrates that the bottom of the screwbosses 320 are positioned a distance above the lens holder body 310. Inthe illustrated lens holder 310, the screw bosses 320 are positioned 0.5mm above the lens holder body 310. The specific distance that the screwbosses 320 are positioned above the lens holder body 310 is typicallydependent upon the requirements of a given application. FIG. 3Cillustrates that almost all of the deformation is experienced at thespider ribs 330 (e.g. stresses shown in dark colored area), with littleto no deformation occurring in the lens holder body 310. Designs thatconnect the lens holder body 310 directly to the screw bosses 320without the spider ribs 330 often experience significant amounts ofdeformation throughout the lens holder body 310, thereby deforming thelens 140 being held within the body 310. By using the spider ribs 330,the lens holder 310 is able to redirect the force exerted from screwingthe lens holder 310 onto the spider legs 330 while maintaining thestructure of the lens holder body 310. Although a range of lens holderdesigns with spider ribs are described above with reference to FIGS.3A-3C, any of a variety of spider rib design configurations can beconstructed, including three or more spider ribs positioned at differentlocations along the lens holder body and at different heights relativeto the screw bosses.

Although the mobile robot 10 is shown with a single camera 125 embeddedwithin the top cover of the mobile robot body 100 in FIGS. 1A-2B, themobile robot 10 can include any of a variety of optional cameraconfigurations, including (but not limited to) one or more cameras 125positioned at different locations along the mobile robot body 100 and atone or more acute viewing angles.

Mobile Robot with Forward and Backward Cameras

The mobile robot 10 may optionally include multiple cameras distributedaround the body 100 of the mobile robot. A particularly advantageousconfiguration involves the use of a tilted forward facing camera 125 anda tilted backward facing camera 410. The forward and backward facingtilted cameras 125, 410 can optionally be contained within separaterecesses 130 a, 130 b within the top 108 of the body 100 of the mobilerobot 10 and protected using lens covers 135 a, 135 b configured in amanner similar to those described above with respect to the mounting ofa single recess 130 and tilted camera 125 behind a lens cover 135 withreference to FIGS. 1A and 2A-2B. The mobile robot 10 configured withforward and backward facing tilted cameras 125, 410 contained withinseparate recesses 130 a, 130 b within the top 108 of the body 100 of themobile robot 10 and protected using lens covers 135 a, 135 b isillustrated in FIG. 4. In particular, FIG. 4 illustrates the mobilerobot 10 configured with a front facing tilted camera 125 aimed in adirection that captures the environment in front of the mobile robot 10and a rear facing tilted camera 410 aimed in a direction that capturesthe environment behind the mobile robot 10. Much like the embodimentdescribed above with regard to FIG. 22, the cameras 125, 410 are angledat about 30 degrees (e.g. 25 degrees, 35 degrees) to focus anapproximately 50 degree high frustum (e.g. 45 degrees, 55 degrees) fieldof view on static features located in the range of 3-8 feet high. Inembodiments, the cameras focus an approximately 60 degree high frustum(e.g., 55 degrees, 65 degrees) field of view on static features locatedin the range of 3-8 feet high. The front and rear cameras 125, 410 areembedded within separate recessed structures 130 a, 130 b within themobile robot body 100 and are each covered by a respective lens cover135 a, 135 b. Each lens cover 135 a, 135 b is set back from the openingof its corresponding recessed structure 130 a, 130 b containing thetilted camera 125, 410 and is aligned at an acute angle α relative tothe plane (e.g., the top surface 108) of the opening of the recessedstructure 130 a, 130 b.

A cross-sectional view of the front and rear facing tilted cameras 125,410 of the mobile robot 10 configured in the manner illustrated in FIG.4 is conceptually illustrated in FIG. 5. The tilted cameras 125, 410 arepositioned such that the respective optical axis 155 a, 155 b of each ofthe cameras 125, 410 is aligned at an acute angle φ above the topsurface 108 in the forward and rearward directions of motion of themobile robot 10. By setting the cameras 125, 410 at these angles, thefront facing camera 125 is primarily aimed at the walls, ceiling, topsof largely immovable furniture such as TV armoires, couches andcountertops directly in front of the mobile robot 10 within a typicalindoor environment and the rear facing camera 410 is primarily aimed atthe walls, ceiling tops of largely immovable furniture such as TVarmoires, couches and countertops behind the mobile robot 10 as ittravels in the forward direction. The use of tilted front and rearfacing cameras 125, 410 allows the mobile robot 10 to observe a highconcentration of reliable, static, and unchanging features to buildrepeatedly recognizable uniquely identifiable landmarks within thesurrounding environment for use in VSLAM while maintaining an angularprecision with which these features are detected. Furthermore, thetilted rear camera 410 may allow the VLSAM process to use the samelandmarks previously used by the tilted front camera 125 when theselandmarks are no longer within the field of view of the front camera125. This may be particularly useful, for example, when the mobile robot10 is configured to navigate its environment in a “corn-row” pattern andwhen the mobile robot 10 resumes localizing following a kidnapping eventin which it is forcibly moved to a new location or loses track of posebecause of lighting variation. In this situation, every time the mobilerobot 10 turns around, the same landmarks that were observed using thefront camera 125 are captured by the rear facing camera 410 to navigateon the return path. Furthermore, the rear facing camera 410 may detectthe landmarks with a higher angular precision than if the mobile robot10 were to use, for example, a 360 degree omnidirectional camera,assuming a same sensor resolution. This is due to the increased spatialresolution that may be attained with the narrowed lens when capturingfeatures within the surrounding environment.

In some embodiments, both cameras 125, 410 may capture images of thesurrounding environment and provide these images to a VSLAM process. Incertain embodiments, only one of the cameras 125 or 410 provides a VSLAMprocess with input images. For example, the mobile robot 10 may use thefront facing camera 125 to detect and track a set of features associatedwith a landmark while moving in a forward direction toward the landmarkand, upon switching directions, use the rear facing camera 410 to detectand track the same set of features while moving away from the landmark.

The mobile robot 10 may simultaneously capture images of the surroundingenvironment using both the tilted front and rear cameras 125, 410,thereby capturing a larger portion of the surrounding environment inless time than a single-camera 125 enabled robot 10. The mobile robot 10may optionally utilize a wide angle, omnidirectional, panoramic orfish-eye type lens to capture more of the surrounding environment at theexpense of a decreased angular resolution. However, by using two cameras125, 410 each with a narrowed field of view in comparison to, forexample, a panoramic camera, to provide input images, the VSLAM processis able to detect a similar number of features as would be achievedusing a panoramic, or similar wide field of view lens, but each featureis captured at a higher angular resolution with a narrower field of viewlens (assuming comparable sensor resolution). In particular, thenarrowed field of view spans a frustum of approximate 50-60 degrees inthe vertical direction and is able to detect features in the environmentat a height of generally 3-14 feet. As is discussed below, providing aVSLAM process with higher precision measurements of the locations offeatures visible within images captured by the machine vision sensorsystem 120 enables the VSLAM process to map the environment and localizethe location of the mobile robot 10 with precision.

Although various optional configurations of the mobile robot 10involving tilted front and rear facing cameras 125, 410 are describedabove with respect to FIGS. 4 and 5, the mobile robot 10 can optionallybe configured using any of a variety of camera configurations includinga tilted front camera 125 combined with a forward camera (not shown)that is aligned in the direction of motion, multiple front facingcameras tilted at different angles, a stereo pair of cameras, two ormore tilted cameras having adjacent or partially overlapping fields ofview, and/or front and rear cameras 125, 140 angled at different anglesto accommodate, for example, a sloped top surface 108 of a robot 10. Theprocess of performing VSLAM using image data captured by one or moretilted cameras 125 in the navigation system 120 of the mobile robot 10is typically performed by a controller circuit 605, which may also beresponsible for implementing other behaviors supported by the mobilerobot 10. Robot controllers 605 and VSLAM processes performed by robotcontrollers in accordance with various embodiments of the invention arediscussed further below.

Robot Controller

The behavior of a mobile robot 10 is typically selected from a number ofbehaviors based upon the characteristics of the mobile robot's 10surrounding operating environment and/or the state of the mobile robot10. In many embodiments, characteristics of the environment may beascertained from images captured by a navigation system 120. Capturedimages can be used by one or more VSLAM processes to map the environmentsurrounding the mobile robot 10 and localize the position of the mobilerobot 10 within the environment.

A mobile robot controller circuit 605 (hereafter “controller circuit605”) that can be used for VSLAM using an enhanced navigation system 120is illustrated in FIG. 6. The robot controller circuit 605 includes aprocessor 610 in communication with a memory 625, a network interface660 and an input/output interface 620. The processor 610 can be a singlemicroprocessor, multiple microprocessors, a many-core processor, amicrocontroller, and/or any other general purpose computing system thatcan be configured by software and/or firmware. The memory 625 contains avisual measurement application 630, a SLAM application 635, one or moremaps of landmarks 640, a behavior control application 645 and alandmarks database 650. The memory 625 can optionally contain any of avariety of software applications, data structures, files and/ordatabases as appropriate to the requirements of specific applications.

The landmarks database 650 contains information concerning a number ofpreviously observed landmarks that the mobile robot 10 can utilize toperform visual measurements from which a relative pose can bedetermined. A landmark can be considered to be a collection of featureshaving a specific 3D structure. Any of a variety of features can beutilized to identify a landmark including (but not limited to) 2Dfeatures, 3D features, features identified using Scale-invariant FeatureTransform (SIFT) descriptors, features identified using Speeded UpRobust Features (SURF) descriptors, and/or features identified usingBinary Robust Independent Elementary Features (BRIEF) descriptors. Whenthe mobile robot 10 is configured as a housecleaning robot, a landmarkcould be (but is not limited to) a set of features identified based uponthe 3D structure of the corner of a picture frame or a set of featuresidentified based upon the 3D structure of a doorframe. Such features arebased on static geometry within the room and, although the features havesome illumination and scale variation, they are more readily discernedand identified in aggregate as landmarks than objects located within alower region of the environment that are frequently displaced (e.g.chairs, trash cans, pets, etc.) In implementations, the camera 125 onthe mobile robot 10 is a 320×240 QVGA, 0.0768 MP camera (or 640×480 VGP,0.3MP camera) that has no IR filter for better detection of features inlow illumination environments. In implementations, particularly when therobot 10 is starting a new mission without storing data between runs orentering a previously unexplored area, the mobile robot 10 will createnew landmarks. In implementations, the mobile robot 10 also will createnew landmarks if lighting variations make previously viewed featuresindiscernible and the number of detectable features falls below athreshold for a minimum number of features for detecting a landmark. Inembodiments, the threshold number of landmarks is a cluster ofidentifiable features detected at a rate 1-10 landmarks per foot oftravel and preferably 3 landmarks per foot of robot travel at a rate ofapproximately 1ft per second or approximately 306 mm per second. Therobot 10 thus builds a useful localization map for features discernableat that lighting intensity and, in implementations, the robot stores oneor more persistent maps with landmarks viewed at various lightintensities, for example those associated with data include a time ofday and calendar date associated with seasonal lighting variations. Instill other implementations, if the environment is too dark and theillumination is insufficient for feature detection, the mobile robot 10will depend on another sensor or combination sensors, such as an wheelodometry and optical dead reckoning drift detection sensor 114 (FIG. 1B)aimed at the floor (e.g. an LED or laser illuminated mouse sensor) tolocalize. In one implementation, the landmarks database 650 includes alandmark image captured from a specific pose that can be referred to asthe landmark pose. A visual measurement involves determining a poserelative to the landmark pose. In order to facilitate visualmeasurement, the landmarks database 650 stores a set of featuresassociated with each landmark and the 3D structure of the set offeatures.

The visual measurement application 630 matches a portion of an inputimage to the landmark image and then determines relative pose based uponthe spatial relationship of the features from the landmark identified inthe input image and the 3D structure of the identified features from thelandmark retrieved from the landmarks database 650. A variety of optionsexist for determining relative pose based upon the features from thelandmark identified in the input image and the 3D structure of theidentified features including (but not limited) determining relativepose based upon the rigid transformation of the 3D structure that yieldsa spatial relationship of features most similar to that observed in theinput image, thereby minimizing reprojection error. Alternatively oradditionally, the rigid transformation yields an estimation of the mostlikely relative pose given the observed spatial relationship of featuresand knowledge of the statistical characteristics of the sources of errorwithin the visual measurement system. Irrespective of the specificprocess utilized to determine the relative pose, the precision of therelative pose estimate is increased by more precisely measuring the 3Dstructure of the features forming a landmark and/or the spatialrelationship between the features identified within an input image.Processes for creating new landmarks for use in visual measurements andfor determining relative pose using landmarks are discussed furtherbelow.

Referring again to FIG. 6, the SLAM application 635 estimates thelocation of the mobile robot 10 within a map of landmarks based upon aprevious location estimate, odometery data, and at least one visualmeasurement received from the visual measurement application. As notedabove, the visual measurement utilized by a SLAM application 635 canoptionally be a relative pose estimate that is determined relative to alandmark pose associated with a landmark identified within the landmarkdatabase 650. In embodiments, the SLAM application 635 uses the relativepose and odometery data and/or mouse sensor drift data to update thelocation estimate of the robot 10 relative to the map of landmarks. Themobile robot 10 updates a location estimate utilizing any of a varietyof SLAM including (but not limited to) the processes described in U.S.Patent Publication 2012/0121161, entitled “Systems and Methods for VSLAMOptimization”, published May 17, 2013, and U.S. Patent Publication2011/0167574, entitled “Methods and Systems for Complete Coverage of aSurface By An Autonomous Robot”, published Jul. 14, 2011, the relevantdisclosures of which are hereby incorporated by reference in theirentirety. The SLAM application 635 can then update the map of landmarks640 based upon the newly estimated location of the mobile robot 10.

The map of landmarks 640 includes a map of the environment surroundingthe mobile robot 10 and the position of landmarks relative to thelocation of the mobile robot within the environment. The map oflandmarks 640 may include various pieces of information describing eachlandmark in the map, including (but not limited to) references to datadescribing the landmarks within the landmark database.

The behavioral control application 630 controls the actuation ofdifferent behaviors of the mobile robot 10 based on the surroundingenvironment and the state of the mobile robot 10. In some embodiments,as images are captured and analyzed by the SLAM application 635, thebehavioral control application 645 determines how the mobile robot 10should behave based on the understanding of the environment surroundingthe mobile robot 10. The behavioral control application 645 may selectfrom a number of different behaviors based on the particularcharacteristics of the environment and/or the state of the mobile robot10. The behaviors may include, but are not limited to, a wall followingbehavior, an obstacle avoidance behavior, an escape behavior, among manyother primitive behaviors that may be actuated by the mobile robot 10.

In several embodiments, the input/output interface 620 provides devicessuch as (but not limited to) sensors with the ability to communicatewith the processor and/or memory. In some embodiments, the networkinterface 660 provides the mobile robot 10 with the ability tocommunicate with remote computing devices, such as computers andsmartphone devices, via a wired and/or wireless data connection.Although various robot controller 605 architectures are illustrated inFIG. 6, any of a variety of architectures including architectures wherethe robot behavioral controller application 645 is located innon-volatile solid state memory or some other form of storage and isloaded into memory at runtime and/or where the robot behavioralcontroller application is implemented using a variety of software,hardware, and/or firmware can be utilized in the implementation of therobot controller circuit 605. The conceptual operation of the mobilerobots 10, when configured by a robot controller similar to the robotcontroller circuit 605 described above is discussed further below.

Mobile Robot Behavioral Control Systems

The mobile robot 10 may include behavioral control applications 710 usedto determine the mobile robot's behavior based upon the surroundingenvironment and/or the state of the mobile robot. The mobile robot 10can include one or more behaviors that are activated by specific sensorinputs and an arbitrator determines which behaviors should be activated.Inputs can include images of the environment surrounding the mobilerobot 10 and behaviors can be activated in response to characteristicsof the environment ascertained from one or more captured images.

A mobile robot behavioral control application 710 configured to enablenavigation within an environment based upon (but not limited to) a VSLAMprocess is conceptually illustrated in FIG. 7. The mobile robotbehavioral control application 710 can receive information regarding itssurrounding environment from one or more sensors 720 (e.g., machinevision system, bump, proximity, wall, stasis, and/or cliff sensors)carried by the mobile robot 10. The mobile robot behavioral controlapplication 710 can control the utilization of robot resources 725(e.g., the wheel modules) in response to information received from thesensors 760, causing the mobile robot 10 to actuate behaviors, which maybe based on the surrounding environment. For example, when the mobilerobot 10 is used to clean an environment, the mobile robot behavioralcontrol application 710 may receive images from a navigation system 120and direct the mobile robot 10 to navigate through the environment whileavoiding obstacles and clutter being detected within the images. Themobile robot behavioral control application 710 can be implemented usingone or more processors in communication with memory containingnon-transitory machine readable instructions that configure theprocessor(s) to implement programmed behaviors 730 and a controlarbitrator 750.

The programmed behaviors 730 can include various modules that may beused to actuate different behaviors of the mobile robot 10. Inparticular, the programmed behaviors 730 may include a VSLAM module 740and corresponding VSLAM database 744, a navigation module 742, and anumber of additional behavior modules 743.

The VSLAM module 740 manages the mapping of the environment in which themobile robot 10 operates and the localization of the mobile robot withrespect to the mapping. The VSLAM module 740 can store data regardingthe mapping of the environment in the VSLAM database 744. The data mayinclude a map of the environment and characteristics of differentregions of the map including, for example, regions that containobstacles, other regions that contain traversable floor, regions thathave been traversed, frontiers to regions that have not yet beentraversed, the date and time of the information describing a specificregion, and/or additional information that may be appropriate to therequirements of a specific application. In many instances, the VSLAMdatabase 744 also includes information regarding the boundaries of theenvironment, including the location of stairs, walls, and/or doors. Ascan readily be appreciated, many other types of data may optionally bestored and utilized by the VSLAM module 740 in order to map theoperating environment of the mobile robot 10. Where the VSLAM module 740performs visual measurements and uses the visual measurements to providerelative poses as inputs to a SLAM module, the VSLAM database 744 caninclude a landmarks database similar to the landmarks database 650described above.

The navigation module 742 actuates the manner in which the mobile robot10 is to navigate through an environment based on the characteristics ofthe environment. For example, in implementations, the navigation module742 may direct the mobile robot 10 to change directions, drive at aspeed of approximately 306 mm per second and then slow down uponapproaching an obstacle, drive in a certain manner (e.g., wigglingmanner to scrub floors, or a pushing against a wall manner to cleansidewalls), or navigate to a home charging station.

Other behaviors 743 may also be specified for controlling the behaviorof the mobile robot 10. Furthermore, to make behaviors 740-743 morepowerful, it is possible to arbitrate between and/or chain the output ofmultiple behaviors together into the input of another behavior module toprovide complex combination functions. The behaviors 740-743 areintended to implement manageable portions of the total cognizance of themobile robot 10.

Referring again to FIG. 7, the control arbitrator 750 facilitatesallowing modules 740-743 of the programmed behaviors 730 to each controlthe mobile robot 10 without needing to know about any other behaviors.In other words, the control arbitrator 750 provides a simple prioritizedcontrol mechanism between the programmed behaviors 730 and resources 725of the robot. The control arbitrator 750 may access behaviors 740-743 ofthe programmed behaviors 730 and control access to the robot resources760 among the behaviors 740-743 at run-time. The control arbitrator 750determines which module 740-743 has control of the robot resources 760as required by that module (e.g. a priority hierarchy among themodules). Behaviors 740-743 can start and stop dynamically and runcompletely independently of each other. The programmed behaviors 730also allow for complex behaviors that can be combined together to assisteach other.

The robot resources 760 may be a network of functional modules (e.g.,actuators, drives, and groups thereof) with one or more hardwarecontrollers. The commands of the control arbiter 750 are typicallyspecific to the resource to carry out a given action. The specificresources with which the mobile robot 10 is configured typically dependsupon the requirements of the specific application to which the mobilerobot 10 is adapted.

Although specific robot controllers and behavioral control applicationsare described above with respect to FIGS. 6-7, any of a variety of robotcontrollers can optionally be utilized within the mobile robot 10including controllers that do not rely upon a behavioral based controlparadigm. Referring again to FIGS. 1-4, the mobile robot 10 includes anavigation system 120 that can include one or more tilted cameras 125.The manner in which the mobile robot 10 can optionally perform VSLAMprocesses in furtherance of its programmed behaviors using input imagescaptured by the tilted camera(s) 125 is discussed further below.

VSLAM Overview

The mobile robot 10 can continuously detect and process information fromvarious on board sensors in order to navigate through an environment. Aprocess 800 that can optionally be used by the mobile robot 10 tonavigate an environment is illustrated in FIG. 8. The process determines(805) a robot behavior. The robot behavior may be determined by a robotcontroller circuit 605 based on various factors, including theparticular characteristics of the environment surrounding the mobilerobot 10, the current state of the mobile robot 10, the particularoperation being executed by the mobile robot, and the state of the powersource used by the mobile robot 10, among various other considerations.The mobile robot 10 determines (810) whether the behavior involves achange in a pose of the robot 10. When the mobile robot 10 determinesthat no change in pose is required, the process completes the behaviorand determines a new behavior (805). Otherwise the process actuates(815) the drive 111 of the mobile robot 10. In some embodiments, thedrive 111 uses two or more wheels 113 a, 113 b to move the robot acrossa surface. In other embodiments, the drive 111 may use one or morerotating pads or tank like tracks that cause the mobile robot 10 to moveacross the surface based on the rotation of the pads or tracks along thesurface.

As the mobile robot 10 moves, image and odometry data is captured (820).In one optional aspect of the invention, the mobile robot 10 captureseach new image after travelling a threshold distance following captureof a prior image, such as (but not limited to) 20 cm between images. Thespecific distance between image captures typically depends on factorsincluding (but not limited to) the speed of the robot 10, the field ofview of the camera 125, and the real time processing capabilities of thespecific mobile robot 10 configuration. In several embodiments, theodometry data is provided by one or more different types of odometers,including a wheel odometer that captures the odometry data based on therotation of the wheels or an optical flow odometry system that obtainsthe odometry data by capturing images of a tracking surface anddetermining distance traveled, including correcting for any headingdrift, based on the optical flow observed between successive images.Other embodiments may use additional odometry sensors or combinations ofthese sensors as appropriate to the requirements of the specificapplication.

Visual measurements can be generated (825) by the mobile robot 10 basedon the odometry data and the captured images captured by the tiltedcamera 125. In some embodiments, the process 800 matches the new imageto a set of landmark images stored in a landmarks database and for eachmatch, estimates a relative pose determined relative to the landmarkpose of the landmark given the 3-D structure and the featurecorrespondences between the new view and the landmark image.

A SLAM process can then be performed (830) using the visual measurementdata and the odometry data. The mobile robot 10 can optionally maintaina map of landmarks and performs the SLAM process to estimate thelocation of the mobile robot 10 within this map. The SLAM process canalso update the map of landmarks.

A determination (835) is made by the mobile robot 10 concerning whetherthe process has completed, and if so, completes. Otherwise, the processdetermines a new behavior (805).

The performance of a navigation process similar to the process describedabove with reference to FIG. 8 can be significantly enhanced byutilizing a camera 125 tilted so that its optical axis 155 forms anacute angle with the forward direction of motion of the mobile robot 10,as described above with reference to FIGS. 2A, 2B and 22. Similarly,increasing the angular resolution of the camera 125 (by utilizing anarrow field of view camera and/or increasing the resolution of thecamera sensor) can increase the effectiveness of the navigationprocesses. The sensitivity of the mobile robot's 10 navigation processesto observed disparity and/or spatial resolution is discussed furtherbelow with reference to the measurement of 3D structure during landmarkcreation and the measurement of the spatial relationship of featuresidentified within input images captured during navigation.

Landmark Creation and Landmark Based Navigation

The mobile robot 10 can utilize landmarks, which are collections offeatures having specific visually discernable 3D structures, to performnavigation. The mobile robot 10 creates landmarks by capturing images ofan environment and observing and aggregating common features between theimages. By overlapping the images and measuring the disparity betweeneach of the features and estimating changes in pose between the captureof the images, the mobile robot 10 moving at a known speed can measurethe distance to each of the features. These distances can then beutilized to determine the 3D structure of the set of features thatdefine a landmark. As is discussed above, when some or all of thefeatures forming the landmark are subsequently observed, knowledge ofthe 3D structure of the features can be utilized to estimate therelative pose of the mobile robot 10. In implementations, the mobilerobot 10 can take two or more images over a distance traveled in orderto localize.

A process that can optionally be utilized by the mobile robot 10 toidentify a new landmark for use in navigation is illustrated in FIG. 9.The process 900 includes obtaining (902) a sequence of input images. Theimage sequence can include as few as two input images. Correspondencescan be established (904) between features in two or more of the inputimages. Any of a variety of features can be utilized including (but notlimited to) 2D features, 3D features, features identified usingScale-invariant Feature Transform (SIFT) descriptors, featuresidentified using Speeded Up Robust Features (SURF) descriptors, and/orfeatures identified using Binary Robust Independent Elementary Features(BRIEF) descriptors. In one implementation, the robot 10 uses FAST SLAMand Brief descriptors. Putative correspondences can be generated usingonly the feature descriptors, or by taking advantage of motion estimatesprovided by odometry data. The process of establishing (904)correspondences can optionally involve applying geometric constraints toeliminate features that are incorrectly identified as correspondingbetween the two or more input images.

The 3D structure of the identified features can then be determined (906)by minimizing the reprojection error between the locations of theobserved features in each of the input images and the predictedlocations given a specific estimate for the 3D structure of theidentified features. Due to uncertainty in the relative poses of themobile robot 10 when the input images were captured, the mobile robot 10can use techniques including (but not limited to) bundle adjustment tosimultaneously determine (906) the 3D structure of the identifiedfeatures and estimate the relative motion of the mobile robot duringcapture of the input images. The mobile robot 10 can optionally use anyof a variety of processes for determining structure from motionincluding (but not limited to) a trifocal tensor method.

Information concerning a newly identified landmark can be added (908) tothe landmarks database. The mobile robot 10 can optionally associate oneor more of the input images or portions of the input images and theestimate of the pose from which a specific input image was captured witha landmark in the landmarks database as landmark images andcorresponding landmark poses. The mobile robot 10 can also optionallystore the features and/or descriptors of the features that areassociated with the landmark and a description of the 3D structure ofthe features. As can readily be appreciated, the specific structure ofthe database is typically determined based upon the requirements of aspecific application and can include (but is not limited to) the use ofa collection of kd-trees for performing efficient approximate nearestneighbor searches based upon observed features. The information added tothe landmarks database by the mobile robot 10 can then be utilized inthe determination of relative poses by the mobile robot 10 duringsubsequent navigation.

A process that can optionally be utilized by the mobile robot 10 todetermine relative pose using previously created landmarks isillustrated in FIG. 10. The process 1000 includes obtaining (1002) oneor more input images. The mobile robot 10 can compare (1004) featureswithin an input image to features associated with various landmarks inthe landmarks database. The process used to identify the features in theinput image is typically the same process utilized to identify featuresduring the creation of new landmarks.

When sufficient similarity exists between features in the input imageand features of one or more landmarks in the landmarks database, themobile robot 10 can determine (1006) that a known landmark from thelandmark database is visible within the input image. The mobile robot 10can then estimate (1008) the pose of the robot relative to a landmarkpose associated with the landmark in the landmarks database based uponthe reprojection of the 3D structure of the features of the landmarkthat most closely matches the spatial relationship of the featuresobserved within the input image in which the landmark is visible. Therelative pose can optionally be a description of the translation and/orrotation of the mobile robot 10 relative to the landmark pose associatedwith the landmark in the landmarks database. The specific manner inwhich the relative pose is represented is largely dependent upon therequirements of a particular application. The process of determining themost likely relative pose estimate can vary depending upon theconfiguration of the mobile robot 10. In one optional configuration ofthe mobile robot 10, a cost metric is utilized to minimize reprojectionerrors of the landmark features visible within an input image. Inanother optional configuration of the mobile robot 10, the estimationprocess considers likely sources of error associated with one or more ofthe process of estimating the 3D structure of the features forming thelandmark, and/or the location of the landmark pose. In one specificconfiguration of the mobile robot 10, an initial estimate of relativepose is formed using reprojection of the 3D structure of the featuresforming the landmark and then a maximum likelihood estimation isperformed assuming that the 3D structure of the features and relativepose can vary. In other configurations of the mobile robot 10, any of avariety of techniques can be utilized to determine relative pose basedupon one or more of odometry data, the location of features in theimages used to create the landmark, the 3D structure of the landmark,and/or the spatial relationship of the features associated with thelandmark that are visible within the input image.

When the mobile robot 10 does not detect the presence of a knownlandmark within an input image, the mobile robot can optionally attemptto create (1010) a new landmark or may simply obtain another inputimage.

The mobile robot 10 can provide (1012) the identity of an identifiedlandmark and the estimated relative pose to a SLAM process, which candetermine a global pose estimate and/or update the global map oflandmarks maintained by the mobile robot. As noted above, a mobile robot10 can be optionally configured using any of a variety of SLAM processesthat rely upon relative pose estimates determined relative to knownlandmarks as inputs. In implementations, the robot 10 uses FAST SLAM andBRIEF descriptors.

The processes described above for creating new landmarks and fornavigating based upon previously created landmarks rely upon the spatialrelationships that can be established using reprojection of the 3Dstructure of a set of features. As discussed further below withreference to FIGS. 11-16B, the alignment of the camera optical axis 155relative to the forward direction of motion of the mobile robot 10 canplay an important role in the precision with which the mobile robot cannavigate. Aligning a camera 125 directly in the direction of forwardmotion of a mobile robot 10 can negatively impact the precision ofreprojection processes. When a camera moves toward a landmark along itsoptical axis 155, little or no disparity is observed and thereprojection process relies upon changes in scale alone. Configuring themobile robot 10 with at least one camera 125 tilted so that its opticalaxis 155 forms an angle above the forward direction of motion of themobile robot 10 can increase observed disparity and the precision ofmeasurements made utilizing reprojection.

An example of the field of view captured by a mobile robot 10 that has acamera 125 configured such that the optical axis 155 is aligned with theforward direction of motion of the mobile robot is illustrated in FIGS.11A-11C. The mobile robot 10 is shown in FIG. 11A within an indoorenvironment. The mobile robot 10 is configured with a forward facingcamera 125 having a field of view that is capturing portions of twowalls 1102, 1104 that meet in a corner 1106 and various portions of theground surface 1108 of the environment. The mobile robot 10 is shownmoving toward the corner 1106 along the optical axis 155 of the camera.Successively smaller portions of the scene 1112, 1114, and 1116 areshown as captured in FIGS. 11A, 11B and 11C. Features that remainvisible in each of the three views 1112, 1114, 1116 shown in FIGS. 11A,11B, and 11C appear larger the closer the mobile robot 10 is to thecorner 1106. However, little disparity is observed within the center ofthe field of view of the camera 125 between the successive views 1112,1114, and 1116. The lack of disparity between views of a feature as acamera 125 moves directly toward the feature in a direction of motionalong the optical axis 155 of the camera 125 is conceptually illustratedin FIG. 13A. As the camera 125 moves closer to the feature, the featureincreases in size in each of three successive views 1300, 1302, 1304.However, the feature remains centered within the field of view of thecamera 125. As such, distance to the feature is difficult to determinewith any precision. Furthermore, features such as corners, which aretypically very distinctive aids to navigation, can appear very similarat different scales. Referring again to FIGS. 11A-11C, the corner 1106will look very similar in each of the views 1112, 1114, and 1116.

As described above with regard to FIGS. 2A, 2B and 22, tilting thecamera 125 so that its optical axis 155 forms an acute angle above thedirection of motion of the mobile robot 10 increases the disparityobserved across the field of view of the camera. When the mobile robot10 is configured as a house cleaning robot, the mobile robot 10 isconstrained to move across a generally planar floor. Even when the flooris sloped, the mobile robot 10 does not move in a direction along theoptical axis 155 of the camera 125. Therefore, translations of themobile robot 10 result in observable disparity between different viewsof the scene.

An example of views of a scene captured by a camera 125 tilted so thatits optical axis 155 forms an acute angle above the direction of motionof the mobile robot 10 are illustrated in FIGS. 12A-12C. The mobilerobot 10 captures views of a similar scene to that shown in FIGS.11A-11C involving two walls 1202, 1204 meeting at a corner 1206. Themobile robot 10 moves along the floor 1208 in a direction of motion 1210toward the corner 1206. As when the mobile robot 10 is configured with aforward looking camera in the manner described above with respect toFIGS. 11A-11C, the mobile robot configured with the tilted camera 125captures views 1212, 1214, 1216 of successively smaller portions of thescene. However, the tilt of the camera 125 results in the field of viewof the camera in each of the successive views shifting lower down thewalls 1202, 1204. Therefore, features visible in each of the three viewsappear to move upward between successive views 1212, 1214, 1216. Theupward movement of features is conceptually illustrated in FIG. 13B. Asthe mobile robot 10 moves along the floor in a direction toward afeature, the feature increases in size and moves progressively upward ineach of three successive views 1310, 1312, 1314.

To illustrate the observable disparity, the three successive views 1310,1312, 1314 are superimposed. When features do not lie on the exactcenterline of the field of view of the camera 125, the features willmove up and away from the centerline of the field of view of the camera.As can readily be appreciated, the presence of significant disparity canbe utilized to determine distance to a feature with much greaterprecision than can be made from the feature observed in the successiveviews 1300, 1302, 1304 illustrated in FIG. 13A. When considered across agroup of features, the increased disparity provides more informationconcerning the 3D structure of a set of features forming a landmark. Forexample, if a set of features corresponds to an object in theenvironment with a 3D structure such as a door frame, rather than just acircle on the wall, then as the mobile robot 10 moves closer to the doorframe, it will be able to ascertain the 3D structure of the doorframewith higher precision. As noted above, the precision with which the 3Dstructure of a landmark is determined significantly impacts theprecision of relative pose estimates made using the landmark duringsubsequent navigation. The tilt of the camera 125 can also enable themobile robot 10 to see more of the underside of objects. Many of theobjects relied upon for navigation by mobile robots 10 configured ashousecleaning objects are relatively flat objects hung or mounted towalls such as (but not limited to) picture frames, flat screentelevisions, and speaker systems. Tilting the camera 125 increases theportion of the field of view of the camera devoted to imaging theunderside of such objects. Accordingly, the 3D structure of theunderside of the objects can be determined more precisely resulting inmore precise subsequent navigation based upon the 3D structure of thelandmark.

The impact of increasing observable disparity when determining the 3Dstructure of a landmark and estimating relative pose is illustrated by aseries of simulations described below with reference to FIGS. 14A, 14B,and 15. The simulations involved simulating the precision with which amobile robot 10 moving toward a corner of a room can determine itsrelative pose using a forward looking camera compared with the precisionachieved using a camera tilted so that the optical axis of the camera125 forms an acute angle above the direction of motion of the mobilerobot 10.

FIG. 14A is an image 1400 generated by simulating a viewpoint of aforward facing camera of a simulated mobile robot in a virtual indoorenvironment. The virtual indoor environment is a room that includestexture on its walls and ceiling in the form of an image of a cheetah1402 and an image of a zebra 1404 on two walls forming a corner and animage of a sunrise 1406 on the ceiling of the virtual room. In thesimulation, the virtual room is dimensioned as a 4 unit by 4 unit roomwith 2 unit high walls. The simulated mobile robot camera is 1 unitabove the floor and the forward looking camera is pointed at a corner.In the simulation, the mobile robot determines shift in relative posebased upon comparisons of features identified in images obtained aftermoving 0.5 units forward toward the corner. The features identified bythe simulated mobile robot using processes similar to those describedabove are illustrated as a plurality of crosses 1408. FIG. 14B is animage 1450 generated by simulating a viewpoint of a camera having thesame field of view as the camera used in the simulation used to generatethe image 1400 shown in FIG. 14A with the exception that the camera istilted so that the optical axis of the camera is at a 30 degree angleabove the direction of forward motion of the simulated mobile robot. Thefeatures identified by the simulated mobile robot and used to performlocalization are illustrated as a plurality of crosses 1452.

FIG. 15 is a chart showing a comparison of the precisions with which themobile robot is able to determine its relative pose in the floor planein each of the simulations described above. The chart 1500 illustratesthe uncertainty 1502 in the relative pose determined using a simulatedmobile robot with a forward facing camera and the uncertainty 1504 inthe relative pose determined using a simulated mobile robot with acamera tilted so that the optical axis of the camera is at a 30 degreeangle above the direction of motion of the mobile robot. As noted above,the simulations involved the simulated mobile robot moving 0.5 unitstoward the corner of the virtual room. Uncertainty is expressed in termsof error to the left or right of the true location of the simulatedmobile robot and in front of or behind the true location of simulatedmobile robot defined relative to the forward direction of motion. Theleft-right uncertainty can be considered uncertainty in a dimensionperpendicular to the direction of motion and the forward-backwarduncertainty can be considered uncertainty in a dimensional parallel tothe direction of motion. As can be appreciated by reviewing FIG. 15, useof a camera tilted so that the optical axis of the camera is at a 30degree angle above the direction of motion of the mobile robot achievesa reduction in uncertainty relative to a forward facing camera in bothdimensions. The reduction in uncertainty in the dimension perpendicularto the direction of motion is slight, however, a significant reductionin uncertainty is achieved in the dimension along the direction ofmotion.

While the simulations described above compare a forward facing cameraand a camera tilted so that its optical axis at a 30 degree angle abovethe direction of motion of the mobile robot, similar reductions inlocation uncertainty can be achieved using cameras tilted so that theoptical axes of the cameras are aligned at other acute angles above thedirection of motion of the mobile robot. Furthermore, although thebenefits of using a tilted camera 125 to increase disparity observed insuccessive images captured by the camera using processes for creatinglandmarks and determining relative pose based upon landmarks aredescribed above with reference to FIGS. 9 and 10, similar benefits canbe obtained by utilizing one or more tilted cameras 125 to captureimages of a scene using the mobile robot 10 configured using any of avariety of optional VSLAM navigation processes.

Cameras employing wide angle lenses are typically utilized in mobilerobots configured to perform VSLAM and CV-SLAM. A significant benefit ofusing a tilted camera 125 is that features observed within the center ofthe field of view of the camera can be utilized to obtain precise depthestimates. As is discussed further below, the precision with which themobile robot 10 can determine the 3D structure of landmarks can befurther increased by utilizing a tilted camera 125 configured with anarrow field of view lens 140 having increased angular resolutionrelative a camera employing a sensor with the same resolution and awider field of view lens.

Increasing Navigation Precision with Increased Angular Resolution

Configuring the mobile robot 10 with a tilted camera 125 having a narrowfield of view and increased angular resolution can increase theprecision with which the mobile robot 10 can determine the 3D structureof landmarks and the precision of relative pose estimates determinedbased upon the reprojection of the 3D structure of the landmarks.Furthermore, the use of narrow field of view lenses can enable themobile robot 10 to perform image processing without the computationalexpense (e.g., time, processing power, etc.) of rectifying the imagesacquired by the camera 125. As can readily be appreciated, decreasingprocessing load can enable the mobile robot 10 to process more images ata higher frame rate (enabling the mobile robot to move faster), decreasepower consumption (increasing battery life), and/or provide the mobilerobot 10 with additional processing capacity to perform other functions.

FIGS. 16A and 16B illustrate the increase in precision with whichlandmarks can be located when the mobile robot 10 is configured using atilted camera 125 with a narrow field of view as compared to when themobile robot is configured using a forward looking camera having acomparatively wider field of view. FIG. 16A illustrates the samplinggrid 1600 of the camera 125 of the mobile robot 10 configured using awide angle lens. The sampling grid 1600 conceptually illustrates theportions of the scene sampled by each pixel in the directly forwardlooking camera 125. In real world applications, the mobile robot 10 istypically configured with a much higher resolution camera. However, avery low resolution sampling grid is utilized to illustrate the impactof angular resolution and distortion on subsequent image processing. Thesampling grid 1600 samples a scene, which is a typical living roomincluding a couch 1620, coffee table 1625, television 1630, framedpicture 1635 and window 1640. When the mobile robot 10 moves in adirection along the optical axis 155 of the camera 125, very littledisparity is observed within the central portion of the field of view ofthe camera and the most reliable depth estimates of 3D structure areobtained from features at the periphery of the field of view of thecamera 125. The distortion introduced by the wide angle lens isconceptually illustrated as curved lines within the sampling grid 1600.Due to distortion, the angular resolution of the camera 125 is lower atthe periphery of the field of view of the camera compared to the angularresolution of the camera at the center of the field of view of thecamera. The lower angular resolution is depicted as greater separationbetween the vertices within the sampling grid 1600. Therefore, featuresobserved within the periphery of the field of view of the camera 125 areobserved with comparatively low precision. A feature that could be usedin navigation is the front corner of the couch. Shading 1610 is used toindicate a grouping of pixels that may be indicative of the landmark. Ascan be seen by comparison with FIG. 16B (discussed below), thecomparatively lower resolution means that significant shifts androtations of the sampling grid 1600 can occur before the feature isobserved using a different set of pixels. Accordingly, the precision oflocalization of the mobile robot 10 is comparatively much lower.

By contrast, the mobile robot 10 illustrated in FIG. 16B samples thescene using a tilted camera 125 having a narrow field of view. Due tothe camera 125 tilt, the mobile robot 10 is able to precisely determinethe 3D structure of features across the field of view of the camera.Narrowing the field of view of the camera 125 significantly increasesthe angular resolution of the camera relative to the angular resolutionof the camera illustrated in FIG. 16A. The sampling grid 1650 of themobile robot 10 illustrated in FIG. 16B samples the scene with muchhigher angular resolution, or sampling density. Therefore, the precisionwith which the location of specific features can be determined is muchhigher. Furthermore, the narrow field of view lens 140 introduces verylittle distortion. Therefore, the angular resolution does not diminishsignificantly across the narrower field of view, and the mobile robot 10can process the image data without first performing rectification. Theability to measure disparity with finer precision using a narrower fieldof view lens can result in more precise estimation of the 3D structureof the features that form a landmark and the distance of thosestructural elements to the mobile robot 10 moving relative to thestationary landmarks at a known speed. In addition, the ability tomeasure the distance between identified features within an image moreprecisely can result in a more precise relative pose estimation basedupon minimization of the reprojection error of the 3D structure into thenew viewpoint. Accordingly, the combination of a tilted camera 125 and anarrow field of view lens 140 can result can result in much higherlocation precision during navigation based upon VSLAM processes. Afeature that could be used in navigation is the top right corner of thepicture frame 1635. Shading 1660 is used to indicate a grouping ofpixels that may be indicative of the landmark. As can be seen bycomparison with FIG. 16A (discussed above), the comparatively higherresolution means that small shifts and/or rotations of the sampling gridwill result in this feature being observed by a different set of pixels.Accordingly, the precision of localization of the mobile robot 10 iscomparatively much higher.

While different fields of view are illustrated in FIGS. 16A and 16B forthe purpose of conceptually illustrating the benefits of utilizing atilted camera 125 with a narrow field of view relative to a forwardlooking camera with a wide angle lens, the mobile robot 10 can beconfigured using a tilted camera 125 having any of a variety of fieldsof view. A narrow field of view lens is typically considered to be alens in which perspective projection is generally a good approximationto the true imaging characteristics of the lens, whereas wide anglelenses introduce distortion. In one optional configuration, the tiltedcamera 125 is configured with a narrow field of view lens having a lenswith a horizontal field of view selected from the range of 65-75 degreesand a vertical field of view selected from the range of 45-65 degrees.In implementations, the tilted camera 125 lens has a 72 degreehorizontal field of view and a 50 degree vertical field of view. Inanother optional configuration, the titled camera 125 is configured witha narrow field of view lens having a field of view of at or between 70and 80 degrees horizontally and 50 and 60 degrees vertically. As canreadily be appreciated, the specific lens 140 and/or the specific fieldof view of the lens 140 utilized in the camera(s) 125 of the mobilerobot 10 is largely dependent upon the requirements of a givenapplication.

The extent of the field of view of the camera 125 utilized by the mobilerobot 10 can become largely irrelevant when some or all of the field ofview of the camera 125 is occluded or severely blurred. Accordingly, themobile robot 10 can be optionally configured to detect occlusions andnotify the user of the need to inspect the mobile robot 10 to attempt todisregard the occlusion from comparisons of successive images. Occlusiondetection processes that can be optionally employed by the mobile robot10 are discussed further below.

Occlusion Detection

The one or more cameras 125, 140 in the navigation system 120 of themobile robot 10 may experience deteriorated functionality due to ongoingexposure to the elements in the environment surrounding the mobile robot10. In particular, within an indoor environment, dust, debris,fingerprints, hair, food particles, and a variety of other objects maycollect and settle on the camera lens cover(s) 135 as the mobile robot10 cleans the environment. These obstructions may diminish the qualityof images captured by the camera 125 for use in a VSLAM process and thusdiminish the accuracy of the mobile robot's 10 navigation through anindoor environment. In order to maintain a sufficient level ofnavigation performance, the mobile robot 10 may provide a notificationto a user when a determination is made that the mobile robot 10 is nolonger receiving useful information from some or all of the view of atleast one camera 125.

The manner in which a portion of the field of view of one of the cameras125 of the mobile robot 10 can become occluded are as varied as theenvironments in which the mobile robot 10 can be configured to operate.When a portion of the field of view is occluded, the camera 125 may notprovide any useful image data that can be used by navigation processesincluding (but not limited to) a VSLAM process. Examples of specifictypes of occlusions and the resulting images captured by the occludedcamera 125 are conceptually illustrated in FIGS. 17A-18B. Asemi-transparent occlusion 1706 is illustrated in FIG. 17A. The camera125 is shown as recessed within the body 100 of the mobile robot 10 andhaving a lens cover 135 located within the opening 1704 of the recess. Asemi-transparent occlusion 1706 (such as a finger smudge or a waterdroplet) is present on the lens cover 135. An image 1710 captured by thecamera 125 in which the semi-transparent occlusion 1706 results in ablurred portion of the image 1712 is conceptually illustrated in FIG.17B. As can readily be appreciated, the blurring is likely to result ina lack of feature correspondences with respect to features that would beotherwise identified within the portion of the camera field of viewdistorted by the semi-transparent occlusion. Therefore, the presence ofthe occlusion 1706 will become evident over time due to an inability ofa navigation process such as (but not limited to) a visual measurementprocess to identify a landmark within the impacted portion of the camerafield of view.

An opaque occlusion 1806 is illustrated in FIG. 18A. The camera 125 issimilarly shown as recessed within the body 100 of the mobile robot 10and having a lens cover 135 located within the opening 1804 of therecess 130. An opaque occlusion 1806 (such as a dust particle, inkdroplet, or fragment of paper) is present on the lens cover 135. Animage 1810 captured by the camera 125 in which the opaque occlusion 1806results in complete occlusion of a portion of the scene 1812 isconceptually illustrated in FIG. 18B. To the extent that the mobilerobot 10 detects a feature within the portion of the scene impacted bythe opaque occlusion 1806, the mobile robot 10 will be unable to observeany disparity with respect to those features in successive views of thescene. The opaque occlusion 1806 typically will appear as a falselandmark in the same location with the same dimensions in successiveimages. This lack of disparity prevents the use of any featuresassociated with the occlusion 1806 in creation of a landmark, becausethe mobile robot 10 would be unable to determine any 3D structure forthe landmark. Accordingly, over time the presence of the opaqueocclusion 1806 will become evident over time due to an inability of anavigation process such as (but not limited to) a visual measurementprocess to identify a landmark within the impacted portion of the camera125 field of view. The mobile robot 10 may optionally detect opaqueocclusions 1806 more rapidly by detecting features that appear in thesame location irrespective of the rotation and/or translation of themobile robot 10.

An occlusion detection process that can optionally be performed by themobile robot 10 is illustrated in FIG. 19. The process 1900 includescapturing (1905) an input image for use in navigation. As noted above,the mobile robot 10 may be optionally configured using a camera 125 thatis located within a recess 130 in the top cover of the body 100 of themobile robot 10 and/or the camera 125 may be tilted so that the opticalaxis 155 of the camera forms an acute angle above the forward directionof motion of the mobile robot 10.

The mobile robot 10 determines (1910) the different portions of thefield of view of the camera(s) 125 in which features are identified andutilized during navigation processes including (but not limited to)VSLAM processes and updates (1915) occlusion detection data accordingly.When the mobile robot 10 is configured in such a way that the navigationprocess performs a VSLAM process, certain portions of an image maycontain features that the VSLAM process may utilize to generatelandmarks for use in performing visual measurements and mapping theenvironment. The mobile robot 10 can collect occlusion detection datadescribing the portions of the images used to generate and/or detectlandmarks, and these portions of the images may correspond to differentportions of the camera field of view. In one optional aspect of theinvention, the mobile robot 10 may maintain a histogram of the variousportions of the field of view being used to identify features. When anocclusion is present, the histogram would reflect that these portions ofthe field of view are not being utilized by VSLAM to generate and/ordetect landmarks.

The mobile robot 10 can determine (1920) whether a certain thresholdnumber of input images have been captured. In many embodiments, thethreshold number of input images may vary and the accuracy of theocclusion detection will generally increase with a larger number ofinput images. When a threshold number of images have not been captured,the process continues (1905) to capture additional images. Inembodiments, the threshold number of images is 1-10 images per foot oftravel and preferably 3 images per foot of robot travel at a rate ofapproximately 1 ft per second or approximately 306 mm per second. Whenthe threshold number of images has been captured, the process determines(1925) whether the occlusion detection data identifies one or moreportions of the field of view of the camera 125 that are capturing imagedata that is not being used by the navigation processes. When theocclusion detection data does not identify one or more portions of thefield of view that are capturing image data that is not being used bythe navigation processes, the mobile robot 10 assumes that the field ofview of the camera(s) 125 is unoccluded, and the occlusion detectionprocess completes.

In one implementation, when the occlusion detection data identifies oneor more portions of the field of view are capturing image data that isnot being used by the navigation processes, the mobile robot 10 provides(1930) a camera occlusion notification to one or more users associatedwith the mobile robot 10 so that the users can clear the occlusion fromthe field of view of the camera 125. In one optional aspect of theprocess, the notification is in the form of an electronic messagedelivered to a user using user account information located on a serverwith which the mobile robot 10 can communicate. For example, a user maybe notified through email, text message, or other communication. Inanother optional aspect, the user may be notified through some form ofindicator on the mobile robot 10 such as a flashing light, sound, orother appropriate alert mechanism. The mobile robot 10 may also beoptionally configured with a wiper blade, fan, air knife, and/or anotherappropriate cleaning mechanism that mobile robot can use to attempt toeliminate the detected occlusion.

A communication diagram illustrating communication between the mobilerobot 10, an external server and a user device in accordance with anoptional aspect of the invention is illustrated in FIG. 20. Inparticular, FIG. 20 illustrates the mobile robot 10 sending anotification of an occlusion to an external server 2020. The mobilerobot 10 may send the notification after it detects the presence of anocclusion that is occluding a portion of the field of view of a cameraon the mobile robot 10 such that the unoccluded portions of cameraimages are no longer useful for localizing the robot 10. In an optionalaspect of the invention, the mobile robot 10 may send the notificationdirectly to a user device using a short distance communication protocolsuch as for example, Bluetooth. Upon receiving a notification from themobile robot 10, the external server 2020 sends a message to one or moreuser devices registered to receive notification. In an optional aspectof the invention, the external server may send any one of an SMS textmessage, an automated voicemail, and/or an electronic mail message. Inanother optional aspect of the invention, the mobile robot 10 may sendthe notification on a periodic basis while detecting the continuedpresence of an occlusion. For example, the mobile robot 10 may send adaily, weekly, or other time based notification reminder while detectingan occlusion. In other optional aspects, the mobile robot 10 may send anotification each time it starts to clean an environment.

A system for notifying user devices of an occlusion in accordance withan embodiment of the invention is illustrated in FIG. 21. The system2100 includes a server 2104 that receives and processes messages fromone or more mobile robots. In particular, when the server receives amessage from the mobile robot 10, it can provide the message to one ormore user devices 2105-2107.

In several embodiments, the variety of user devices can use HTTP, SMStext, or another appropriate protocol to receive message via a network2108 such as the Internet. In the illustrated embodiment, user devicesinclude personal computers 2105-2106 and mobile phones 2107. In otherembodiments, user devices can include consumer electronics devices suchas DVD players, Blu-ray players, televisions, set top boxes, video gameconsoles, tablets, and other devices that are capable of connecting to aserver via HTTP and receiving messages.

While the above contains descriptions of many specific optional aspectsof the invention, these should not be construed as limitations on thescope of the invention, but rather as an example of differentconfigurations thereof. Accordingly, the scope of the invention shouldbe determined not by the examples illustrated, but by the appendedclaims and their equivalents.

1. (canceled)
 2. A mobile robot configured to navigate an operatingenvironment, comprising: a machine vision sensor system comprising acamera; an indicator light; at least one processor; and a memorycomprising computer readable program code stored in a non-transitorymedium, the computer readable program code comprising a simultaneouslocalization and mapping (SLAM) application and a behavioral controlapplication, wherein, when executed by the at least one processor, thecomputer readable program code causes the at least one processor to:capture images of the operating environment of the mobile robot usingthe machine vision sensor system; detect a presence of an occlusionobstructing a portion of a field of view of the camera based on theimages captured using the machine vision sensor system; and illuminatethe indicator light responsive to detection of the presence of theocclusion obstructing the portion of the field of view of the camera. 3.The mobile robot of claim 2, wherein the memory further comprises apre-loaded map of the operating environment stored in the non-transitorymedium, and wherein, when executed by the at least one processor, thecomputer readable program code causes the at least one processor to:navigate the mobile robot in the operating environment based on thepre-loaded map.
 4. The mobile robot of claim 2, further comprising: anodometry sensor system, wherein, when executed by the at least oneprocessor, the computer readable program code causes the at least oneprocessor to: capture odometry data using the odometry sensor system;and estimate a location of the mobile robot within the operatingenvironment based on the odometry data captured using the odometrysensor system responsive to insufficient feature detection using themachine vision sensor system.
 5. The mobile robot of claim 4, whereinthe odometry sensor system comprises a wheel odometer configured tocapture the odometry data based on rotation of wheels of a drivemechanism and/or an optical flow odometry system configured to obtainthe odometry data by capturing images of a tracking surface anddetermining a distance traveled thereon.
 6. The mobile robot of claim 2,wherein the camera comprises an optical axis that is tilted at an anglerelative to a top surface of the mobile robot.
 7. The mobile robot ofclaim 6, wherein the camera is housed within a recess in a body of themobile robot, the body comprising a lens cover aligned at an acute anglewith respect to the optical axis of the'camera.
 8. The mobile robot ofclaim 3, wherein the pre-loaded map comprises boundaries of theoperating environment, the boundaries comprising at least one of stairs,walls, or doors.
 9. The mobile robot of claim 2, further comprising: acleaning mechanism for the camera, wherein, when executed by the atleast one processor, the computer readable program code causes the atleast one processor to operate the cleaning mechanism to attempt toeliminate the occlusion responsive to detection of the presence thereof.10. The mobile robot of claim 9, wherein the cleaning mechanismcomprises at least one of a wiper blade, a fan, or an air knife.
 11. Themobile robot of claim 2, wherein, to detect the presence of theocclusion, the computer readable program code, when executed by the atleast one processor, causes the at least one processor to: identify theportion of the field of view of the camera that is not utilized by theSLAM application for a threshold number of images.
 12. The mobile robotof claim 11, wherein, to identify the portion of the field of view ofthe camera, the computer readable program code, when executed by the atleast one processor, causes the at least one processor to: determinerespective portions of the field of view of the camera in which featuresare identified; update occlusion detection data describing portions ofthe images that correspond to the respective portions of the field ofview of the camera; and identify the portion of the field of view of thecamera that is not utilized by the SLAM application for the thresholdnumber of images based on the occlusion detection data.
 13. The mobilerobot of claim 2, wherein the computer readable program code, whenexecuted by the at least one processor, further directs the at least oneprocessor to identify a percentage of the field of view of the camerathat is occluded prior to illumination of the indicator light.
 14. Amethod of operating a mobile robot, the method comprising: capturingimages of an operating environment of the mobile robot using a machinevision sensor system comprising a camera of the mobile robot; detectinga presence of an occlusion obstructing a portion of a field of view ofthe camera based on the images captured using the machine vision sensorsystem; and illuminating an indicator light of the mobile robotresponsive to detection of the presence of the occlusion obstructing theportion of the field of view of the camera, wherein the capturing, thedetecting, and the illuminating comprise operations performed by atleast one processor of the mobile robot.
 15. The method of claim 14,further comprising: navigating the mobile robot in the operatingenvironment based on a pre-loaded map of the operating environment thatis stored in a memory comprising non-transitory medium.
 16. The methodof claim 14, further comprising: capturing odometry data using anodometry sensor system; and estimating a location of the mobile robotwithin the operating environment based on the odometry data capturedusing the odometry sensor system responsive to insufficient featuredetection using the machine vision sensor system.
 17. The method ofclaim 14, wherein the camera of the mobile robot comprises an opticalaxis that is tilted at an angle relative to a top surface of the mobilerobot.
 18. The method of claim 15, wherein the pre-loaded map comprisesboundaries of the operating environment, the boundaries comprising atleast one of stairs, walls, or doors.
 19. The method of claim 14,further comprising: operating a cleaning mechanism comprising at leastone of a wiper blade, a fan, or an air knife to attempt to eliminate theocclusion responsive to detecting the presence thereof
 20. The method ofclaim 14, wherein detecting the presence of the occlusion comprises:identifying the portion of the field of view of the camera that is notutilized by the SLAM application for a threshold number of images.
 21. Amobile robot configured to navigate an operating environment,comprising: a body containing: a drive mechanism; an indicator light; anodometry sensor system configured to capture odometry data; a machinevision sensor system comprising a camera configured to capture images ofthe operating environment; at least one processor; and a memorycomprising computer readable program code stored in a non-transitorymedium, the computer readable program code comprising a visualsimultaneous localization and mapping (VSLAM) application and abehavioral control application, wherein, when executed by the at leastone processor, the computer readable program code causes the at leastone processor to: detect a presence of an occlusion obstructing aportion of a field of view of the camera based on the images captured bythe machine vision sensor system; flash the indicator light responsiveto detection of the presence of the occlusion obstructing the portion ofthe field of view of the camera; and localize the mobile robot withinthe operating environment based on the odometry data captured by theodometry sensor system responsive to insufficient feature detection bythe machine vision sensor system.